Search Results: "arno"

1 September 2015

Lunar: Reproducible builds: week 18 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Aur lien Jarno uploaded glibc/2.21-0experimental1 which will fix the issue were locales-all did not behave exactly like locales despite having it in the Provides field. Lunar rebased the pu/reproducible_builds branch for dpkg on top of the released 1.18.2. This made visible an issue with udebs and automatically generated debug packages. The summary from the meeting at DebConf15 between ftpmasters, dpkg mainatainers and reproducible builds folks has been posted to the revelant mailing lists. Packages fixed The following 70 packages became reproducible due to changes in their build dependencies: activemq-activeio, async-http-client, classworlds, clirr, compress-lzf, dbus-c++, felix-bundlerepository, felix-framework, felix-gogo-command, felix-gogo-runtime, felix-gogo-shell, felix-main, felix-shell-tui, felix-shell, findbugs-bcel, gco, gdebi, gecode, geronimo-ejb-3.2-spec, git-repair, gmetric4j, gs-collections, hawtbuf, hawtdispatch, jack-tools, jackson-dataformat-cbor, jackson-dataformat-yaml, jackson-module-jaxb-annotations, jmxetric, json-simple, kryo-serializers, lhapdf, libccrtp, libclaw, libcommoncpp2, libftdi1, libjboss-marshalling-java, libmimic, libphysfs, libxstream-java, limereg, maven-debian-helper, maven-filtering, maven-invoker, mochiweb, mongo-java-driver, mqtt-client, netty-3.9, openhft-chronicle-queue, openhft-compiler, openhft-lang, pavucontrol, plexus-ant-factory, plexus-archiver, plexus-bsh-factory, plexus-cdc, plexus-classworlds2, plexus-component-metadata, plexus-container-default, plexus-io, pytone, scolasync, sisu-ioc, snappy-java, spatial4j-0.4, tika, treeline, wss4j, xtalk, zshdb. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: Chris Lamb also noticed that binaries shipped with libsilo-bin did not work. Documentation update Chris Lamb and Ximin Luo assembled a proper specification for SOURCE_DATE_EPOCH in the hope to convince more upstreams to adopt it. Thanks to Holger it is published under a non-Debian domain name. Lunar documented easiest way to solve issues with file ordering and timestamps in tarballs that came with tar/1.28-1. Some examples on how to use SOURCE_DATE_EPOCH have been improved to support systems without GNU date. reproducible.debian.net armhf is finally being tested, which also means the remote building of Debian packages finally works! This paves the way to perform the tests on even more architectures and doing variations on CPU and date. Some packages even produce the same binary Arch:all packages on different architectures (1, 2). (h01ger) Tests for FreeBSD are finally running. (h01ger) As it seems the gcc5 transition has cooled off, we schedule sid more often than testing again on amd64. (h01ger) disorderfs has been built and installed on all build nodes (amd64 and armhf). One issue related to permissions for root and unpriviliged users needs to be solved before disorderfs can be used on reproducible.debian.net. (h01ger) strip-nondeterminism Version 0.011-1 has been released on August 29th. The new version updates dh_strip_nondeterminism to match recent changes in debhelper. (Andrew Ayer) disorderfs disorderfs, the new FUSE filesystem to ease testing of filesystem-related variations, is now almost ready to be used. Version 0.2.0 adds support for extended attributes. Since then Andrew Ayer also added support to reverse directory entries instead of shuffling them, and arbitrary padding to the number of blocks used by files. Package reviews 142 reviews have been removed, 48 added and 259 updated this week. Santiago Vila renamed the not_using_dh_builddeb issue into varying_mtimes_in_data_tar_gz_or_control_tar_gz to align better with other tag names. New issue identified this week: random_order_in_python_doit_completion. 37 FTBFS issues have been reported by Chris West (Faux) and Chris Lamb. Misc. h01ger gave a talk at FrOSCon on August 23rd. Recordings are already online. These reports are being reviewed and enhanced every week by many people hanging out on #debian-reproducible. Huge thanks!

25 August 2015

Lunar: Reproducible builds: week 17 in Stretch cycle

A good amount of the Debian reproducible builds team had the chance to enjoy face-to-face interactions during DebConf15.
Names in red and blue were all present at DebConf15
Picture of the reproducible builds talk during DebConf15
Hugging people with whom one has been working tirelessly for months gives a lot of warm-fuzzy feelings. Several recorded and hallway discussions paved the way to solve the remaining issues to get reproducible builds part of Debian proper. Both talks from the Debian Project Leader and the release team mentioned the effort as important for the future of Debian. A forty-five minutes talk presented the state of the reproducible builds effort. It was then followed by an hour long roundtable to discuss current blockers regarding dpkg, .buildinfo and their integration in the archive. Picture of the reproducible builds roundtable during DebConf15 Toolchain fixes Reiner Herrmann submitted a patch to make rdfind sort the processed files before doing any operation. Chris Lamb proposed a new patch for wheel implementing support for SOURCE_DATE_EPOCH instead of the custom WHEEL_FORCE_TIMESTAMP. akira sent one making man2html SOURCE_DATE_EPOCH aware. St phane Glondu reported that dpkg-source would not respect tarball permissions when unpacking under a umask of 002. After hours of iterative testing during the DebConf workshop, Sandro Knau created a test case showing how pdflatex output can be non-deterministic with some PNG files. Packages fixed The following 65 packages became reproducible due to changes in their build dependencies: alacarte, arbtt, bullet, ccfits, commons-daemon, crack-attack, d-conf, ejabberd-contrib, erlang-bear, erlang-cherly, erlang-cowlib, erlang-folsom, erlang-goldrush, erlang-ibrowse, erlang-jiffy, erlang-lager, erlang-lhttpc, erlang-meck, erlang-p1-cache-tab, erlang-p1-iconv, erlang-p1-logger, erlang-p1-mysql, erlang-p1-pam, erlang-p1-pgsql, erlang-p1-sip, erlang-p1-stringprep, erlang-p1-stun, erlang-p1-tls, erlang-p1-utils, erlang-p1-xml, erlang-p1-yaml, erlang-p1-zlib, erlang-ranch, erlang-redis-client, erlang-uuid, freecontact, givaro, glade, gnome-shell, gupnp, gvfs, htseq, jags, jana, knot, libconfig, libkolab, libmatio, libvsqlitepp, mpmath, octave-zenity, openigtlink, paman, pisa, pynifti, qof, ruby-blankslate, ruby-xml-simple, timingframework, trace-cmd, tsung, wings3d, xdg-user-dirs, xz-utils, zpspell. The following packages became reproducible after getting fixed: Uploads that might have fixed reproducibility issues: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: St phane Glondu reported two issues regarding embedded build date in omake and cduce. Aur lien Jarno submitted a fix for the breakage of make-dfsg test suite. As binutils now creates deterministic libraries by default, Aur lien's patch makes use of a wrapper to give the U flag to ar. Reiner Herrmann reported an issue with pound which embeds random dhparams in its code during the build. Better solutions are yet to be found. reproducible.debian.net Package pages on reproducible.debian.net now have a new layout improving readability designed by Mattia Rizzolo, h01ger, and Ulrike. The navigation is now on the left as vertical space is more valuable nowadays. armhf is now enabled on all pages except the dashboard. Actual tests on armhf are expected to start shortly. (Mattia Rizzolo, h01ger) The limit on how many packages people can schedule using the reschedule script on Alioth has been bumped to 200. (h01ger) mod_rewrite is now used instead of JavaScript for the form in the dashboard. (h01ger) Following the rename of the software, debbindiff has mostly been replaced by either diffoscope or differences in generated HTML and IRC notification output. Connections to UDD have been made more robust. (Mattia Rizzolo) diffoscope development diffoscope version 31 was released on August 21st. This version improves fuzzy-matching by using the tlsh algorithm instead of ssdeep. New command line options are available: --max-diff-input-lines and --max-diff-block-lines to override limits on diff input and output (Reiner Herrmann), --debugger to dump the user into pdb in case of crashes (Mattia Rizzolo). jar archives should now be detected properly (Reiner Herrman). Several general code cleanups were also done by Chris Lamb. strip-nondeterminism development Andrew Ayer released strip-nondeterminism version 0.010-1. Java properties file in jar should now be detected more accurately. A missing dependency spotted by St phane Glondu has been added. Testing directory ordering issues: disorderfs During the reproducible builds workshop at DebConf, participants identified that we were still short of a good way to test variations on filesystem behaviors (e.g. file ordering or disk usage). Andrew Ayer took a couple of hours to create disorderfs. Based on FUSE, disorderfs in an overlay filesystem that will mount the content of a directory at another location. For this first version, it will make the order in which files appear in a directory random. Documentation update Dhole documented how to implement support for SOURCE_DATE_EPOCH in Python, bash, Makefiles, CMake, and C. Chris Lamb started to convert the wiki page describing SOURCE_DATE_EPOCH into a Freedesktop-like specification in the hope that it will convince more upstream to adopt it. Package reviews 44 reviews have been removed, 192 added and 77 updated this week. New issues identified this week: locale_dependent_order_in_devlibs_depends, randomness_in_ocaml_startup_files, randomness_in_ocaml_packed_libraries, randomness_in_ocaml_custom_executables, undeterministic_symlinking_by_rdfind, random_build_path_by_golang_compiler, and images_in_pdf_generated_by_latex. 117 new FTBFS bugs have been reported by Chris Lamb, Chris West (Faux), and Niko Tyni. Misc. Some reproducibility issues might face us very late. Chris Lamb noticed that the test suite for python-pykmip was now failing because its test certificates have expired. Let's hope no packages are hiding a certificate valid for 10 years somewhere in their source! Pictures courtesy and copyright of Debian's own paparazzi: Aigars Mahinovs.

1 October 2014

Vincent Sanders: It is a bad plan that admits of no modification

I find it somewhat interesting that thousands of years later that our society still uses Publilius Syrus sententiae though I imagine the tendency to leave well enough alone means such phrases stay in usage.

Marvell ARM system - Photo from Steve McIntyre
One weekend Steve McIntyre asked me if I could find a source of some of some 40mm fans for some systems with some pretty strict requirements. They needed to be long life and shift a lot of air to combat a persistent overheating issue.

I sat with him and went through the Farnell utterly hateful parametric web interface and eventually came up with a couple of options which were very expensive. Only then did I stop and ask what the actual problem was.

Marvell ARM system Original internal cooling arrangement - Photo from Steve McIntyre
Steve showed me one of the Debian ARM buildd boxes which are Marvell development machines. These systems are powerful quad core machines housed in compact steel enclosures.

There is a single 40mm fan trying to provide cooling for the entire enclosure. When the units are placed horizontally and used intermittently this proves adequate. Unfortunately when the system are arranged vertically in a rack and run at full load continuously they often overheat and have to be restarted. In addition the small high speed fans need replacing frequently as their bearings wore out quickly.

Debian ARM buildd systems - Photo from Steve McIntyreThis was obviously causing some issues for the ARM Debian ports which Steve wanted to rectify. After talking the problem through for a while we came to the conclusion we could use much larger 60mm fans to blow air directly through the top of the case onto the cpu heatsink.

Larger fans can be run much more slowly to move a similar volume of air to the smaller 40mm fans which gives a much longer service life.

Hole punch and Drilling template
Steve proceeded to order enough parts to allow us to modify all the Debian systems, this worked out cheaper than a single "special" 40mm high volume fan.

I acquired a rather large steel hole punch, I chose this tool because it produces a much superior finish to a hole cutter and this project demanded a high level of finish (not to mention I loved having a valid excuse to own and use a huge allen key!)

If we had simply been modifying a single case I would have measured and marked up by hand. With the prospect of altering at least eight I laser cut a template from plywood which Andy Simpkins took great glee in excessively annotating.

We also used the opportunity to add bolt holes to securely attach the 2.5 inch SATA drives instead of using sticky pads.

Steve and I modified a single system to begin with both to check our alignment and the efficacy of the change. We were pleasantly surprised to discover that hoiby could now repeatedly do kernel compiles with all four cores flat out which was not possible before. The measured CPU temperature, which had previously been around 90 C, did not rise above 40 C

Steve and Andy on the assembly line
Steve, Andy and I then arranged a day where we took all the remaining units out of the rack at ARM, modified and returned them. We used the facilities at the Cambridge Makespace where I am a member to do the modifications.

I broke two 3mm drill bits and dulled a 4mm bit drilling all the holes, Roger Smith was good enough to loan us the use of his "Christmas tree bit" to ream the fan hole out to 16mm so we could thread the hole punch and cut the 60mm fan aperture out.

six modified systems ready to be re-racked.
We managed to get quite an assembly line going and, in my opinion, the results look pretty professional.

It has been several months since we did this work and these systems continue to run without issue. To complete the story we can see some graphs courtesy of the DSA munin instance.

CPU load on arnold.debian.org
You can clearly see the huge drop in temperature at the end of Week 25 despite the continuously high CPU load. Also there is only a single gap in the data after the changes (these indicate crashes where data was not recorded) where before there were frequent and extensive times where the systems were simply unusable.

CPU Temperature of arnold.debian.org
One reason I continue to enjoy Debian so much is the wide variety of ways in which I can contribute not only by maintaining my packages. Sometimes this kind of work does not receive the credit it deserves and hopefully highlights a small part of the frantic paddling that goes on under the serene surface of the Debian project to keep things "just working".

20 August 2014

Aurelien Jarno: MIPS Creator CI20

I have received two MIPS Creator CI20 boards, thanks to Imagination Technologies. It s a small MIPS32 development board: mips-ci20 As you can see it comes in a nice packaging with a world-compatible power adapter. It uses a Ingenic JZ4780 SoC with a dual core MIPS32 CPU running at 1.2GHz with a PowerVR SGX540 GPU. The board is fitted with 1GB of RAM, 8GB of NOR flash, HDMI output, USB 2.0 ports, Ethernet + Wi-Fi + BlueTooth, SD card slot, IR receiver, expansion headers and more. The schematics are available. The Linux kernel and the U-Boot bootloader sources are also available. Powering this board with a USB keyboard, a USB mouse and a HDMI display, it boots off the internal flash on a Debian Wheezy up to the XFCE environment. Besides the kernel, the Wi-Fi + Bluetooth firmware, and very few configuration changes, it runs a vanilla Debian. Unfortunately I haven t found time to play more with it yet, but it looks already quite promising. The board has not been formally announced yet, so I do not know when it will become available, nor the price, but if you are interested I ll bring it to DebConf14. Don t hesitate to ask me if you want to look at it or play with it.

15 August 2014

Aurelien Jarno: Intel about to disable TSX instructions?

Last time I changed my desktop computer I bought a CPU from the Intel Haswell family, the one available on the market at that time. I carefully selected the CPU to make sure it supports as many instructions extensions as possible in this family (Intel likes segmentation, even high-end CPUs like the Core i7-4770k do not support all possible instructions). I ended-up choosing the Core i7-4771 as it supports the Transactional Synchronization Extensions (Intel TSX) instructions, which provide transactional memory support. Support for it has been recently added in the GNU libc, and has been activated in Debian. By choosing this CPU, I wanted to be sure that I can debug this support in case of bug report, like for example in bug#751147. Recently some computing websites started to mention that the TSX instructions have bugs on Xeon E3 v3 family (and likely on Core i7-4771 as they share the same silicon and stepping), quoting this Intel document. Indeed one can read on page 49:
HSW136. Software Using Intel TSX May Result in Unpredictable System Behavior

Problem: Under a complex set of internal timing conditions and system events, software using the Intel TSX (Transactional Synchronization Extensions) instructions may result in unpredictable system behavior.
Implication: This erratum may result in unpredictable system behavior.
Workaround: It is possible for the BIOS to contain a workaround for this erratum.
And later on page 51:
Due to Erratum HSw136, TSX instructions are disabled and are only supported for software development. See your Intel representative for details.
The same websites tell that Intel is going to disable the TSX instructions via a microcode update. I hope it won t be the case and that they are going to be able to find a microcode fix. Otherwise it would mean I will have to upgrade my desktop computer earlier than expected. It s a bit expensive to upgrade it every year and that s a the reason why I skipped the Ivy Bridge generation which didn t bring a lot from the instructions point of view. Alternatively I can also skip microcode and BIOS updates, in the hope I won t need another fix from them at some point.

18 June 2014

Aurelien Jarno: Debian is switching (back) to GLIBC

Five years ago Debian and most derivatives switched from the standard GNU C Library (GLIBC) to the Embedded GLIBC (EGLIBC). Debian is now about to take the reverse way switching back to GLIBC, as EGLIBC is now a dead project, the last release being the 2.19 one. At the time of writing the glibc package has been uploaded to experimental and sits in the NEW queue. EGLIBC is dead for a good reason: the GLIBC development has changed a lot in the recent years, due to two major events: Ulrich Drepper leaving Red Hat and the GLIBC development, and the GLIBC steering committe self-dissolving. This has resulted in a much more friendly development based on team work with good cooperation. The development is now based on peer review, which results in less buggy code (humans do make mistakes). It has also resulted in things that were clearly impossible before, like using the same repository for all architectures, and even getting rid of the ports/ directory. Before we used to have two sets of architectures, the main ones in the glibc repository with architectures like x86, SuperH or SPARC, and the secondary ones in the glibc-ports repository with architectures like ARM or MIPS. As you can see the separation was quite arbitrary, and often leaded to missing changes on the secondary architectures. We also got real stable branches, with regular fixes. The most important EGLIBC features have been merged to GLIBC, including for example the use of non-bash but POSIX shell, or the renaming of reserved keywords. The notable exception is the support for configurable components, which we originally planned to use for Debian-Installer, by building a smaller flavor using -Os and without NIS and RPC support. At the end we never worked on that, and it seems that the hardware targeted by Debian has grown faster than the GLIBC size, so that is not really a big loss. At the end, we ended up with only 5 missing patches from the EGLIBC tree: The package names are unchanged (except the source package and the binary package containing the sources) so the transition is fully transparent for the users. I would like to thank all the CodeSourcery employees who worked on EGLIBC, with a special thank to Joseph Myers who spent countless hours to merge the most important EGLIBC changes back to GLIBC, and sent regular emails about the merge status. I would also like to thanks all the people on the GLIBC side that made the change to happen, and all persons participating in the GLIBC development.

15 May 2014

Thorsten Glaser: Debian packaging example: PHP5 webapp with dbconfig-common and Apache 2.2/2.4 support

I m holding a Debian packaging workshop for our trainees at work tomorrow, and have prepared a sample package for a simple PHP web application (just a handful of files) with DB connection (PostgreSQL of course), automatic setup via dbconfig-common, and with support for both Apache 2.2 (wheezy, precise) and Apache 2.4 (jessie/sid), configuration-wise. (It is possible to install this without Apache, just it does not configure the webserver then.) Schema updates on software updates are also tested (there is neither Flyway nor Liquibase which are the tools we use at work for this, other than Roland Mas wonderful scripts for FusionForge in Debian, but to my delight I discovered that dbconfig-common can also do this). Comments, suggestions, flames, etc. welcome. I know that this should not be a native package, and will address this tomorrow, but I wanted something that serves as decent example for how to do this easily, Policy conformant and using modern techniques (even those I dislike myself for the sake of simplicity). Permission was granted by the business administration to reproduce this all under a BSD-style licence, so, enjoy sharing! Thanks to Roland Mas, for making FusionForge such a nice project, and Arno T ll for some instant IRC help on the Apache side of this. This is my first time using dbconfig-common, and now, I finally feel I know enough to finish the packaging of Kivitendo which I ve started earlier. Beta testers for that welcome, too. (And next week or so, I ll need this for a Maven thingy. I ll probably opt out on the DB side, there, though. Never did anything with that, either, not being a Java guy. I guess something web to go with tomcat7 anyone got this already?)

16 February 2014

Aurelien Jarno: On configure systems

I will never understand the point of using autotools, cmake or whatever configure system, when later the code uses an hardcoded list of architectures to determine the size of a pointer Unfortunately for porters this pattern is quite common. Update: As people keep asking, the way to check for the size of a given type is explained in the autoconf manual. To check for the size of the pointer, the following entry has to be added to configure.ac: AC_CHECK_SIZEOF(void *) On a 64-bit system, this will lead to the following entry in config.h:

/* The size of void *', as computed by sizeof. */
#define SIZEOF_VOID_P 8

6 January 2014

Aurelien Jarno: Debian QEMU images updated

Following the release of Debian Wheezy, I have (finally) updated my set of Debian QEMU images for both Squeeze (6.0.8) and Wheezy (7.3). The following images are now available: Each of these directories contains a GPG signed README.txt file with the md5sums all files, detailed instructions how to run these images and especially the minimum QEMU version to use. The requirements to run the default desktop environment have increased a lot between Squeeze and Wheezy. An accelerated graphics card is now needed to be able to use Gnome (unless you use the fallback mode), which is not something provided by QEMU (actually there is now a QXL para-virtualized video card, but the driver is only in unstable). In addition GDM now needs more than 128MiB to start, while this is the default amount of memory provided for virtual machines. I have therefore decided to switch the default desktop environment on the Wheezy images to Xfce and the display manager to LightDM. Both Gnome and GDM are still installed on the images, and the original default can easily be restored using the following commands: Beside this the new images only contain minor changes. The filesystems have been tweaked to not run fsck after a certain amount of days, and locales-all and openssh-server are now installed in all images. For MIPS and MIPSEL, 64-bit kernels are now also installed and provided, so that it is possible to choose between a 32-bit or a 64-bit kernel (see the README.txt for more details). There is no Debian Wheezy SPARC image available, as QEMU does not fully support SPARC64 yet (it is actually possible to run it, but then the VM crashes often), and Debian Wheezy now only supports 64-bit kernels. I will also invest time to build an S390X image, but so far I haven t been successful on that. The following images are still available at the same location, though they haven t been updated:

29 October 2013

Aurelien Jarno: Detecting code issues using multiple architectures

Sometimes building the same code on multiple architectures is useful to detect horrors like:

ExEnv::err0() < < sprintf("AtomInfo: invalid name: %s\n",name.c_str());
 This code builds using -Werror=format-security with only a few warnings on most architectures, while GCC is correctly detects the issue on some others. This has been reported as bug#728249.

29 September 2013

Ian Campbell: qcontrol 0.5.2

I've just released qcontrol 0.5.2. Changes since the last release: Get it from gitorious or http://www.hellion.org.uk/qcontrol/releases/0.5.2/. The Debian package will be uploaded shortly.

30 January 2013

Paul Tagliamonte: dput-ng/1.4 in unstable

Changes: 
dput-ng (1.4) unstable; urgency=low
   [ Arno T ll ]
   * Really fix #696659 by making sure the command line tool uses the most recent
     version of the library.
   * Mark several fields to be required in profiles (incoming, method)
   * Fix broken tests.
   * Do not run the check-debs hook in our mentors.d.n profile
   * Fix "[dcut] dm bombed out" by using the profile key only when defined
     (Closes: #698232)
   * Parse the gecos field to obtain the user name / email address from the local
     system when DEBFULLNAME and DEBEMAIL are not set.
   * Fix "dcut reschedule sends "None-day" to ftp-master if the delay is
     not specified" by forcing the corresponding parameter (Closes: #698719)
 .
   [ Luca Falavigna ]
   * Implement default_keyid option. This is particularly useful with multiple
     GPG keys, so dcut is aware of which one to use.
   * Make scp uploader aware of "port" configuration option.
 .
   [ Paul Tagliamonte ]
   * Hack around Launchpad's SFTP implementation. We musn't stat *anything*.
     "Be vewy vewy quiet, I'm hunting wabbits" (Closes: #696558).
   * Rewrote the test suite to actually test the majority of the codepaths we
     take during an upload. Back up to 60%.
   * Added a README for the twitter hook, Thanks to Sandro Tosi for the bug,
     and Gergely Nagy for poking me about it. (Closes: #697768).
   * Added a doc for helping folks install hooks into dput-ng (Closes: #697862).
   * Properly remove DEFAULT from loadable config blocks. (Closes: #698157).
   * Allow upload of more then one file. Thanks to Iain Lane for the
     suggestion. (Closes: #698855).
 .
   [ Bernhard R. Link ]
   * allow empty incoming dir to upload directly to the home directory
 .
   [ Sandro Tosi ]
   * Install example hooks (Closes: #697767).
Thanks to all the contributors! For anyone who doesn t know, you should check out the docs.

14 December 2012

Paul Tagliamonte: Scripting Python in Clojure

Yeah, wait, what? No, really! I took a look through clojure-py, and found out something pretty rad. In order to get clojurepy scripts to load other clojurepy, it had to shim out and add a new importer for clojure scripts. What this really means, is that you can import Clojure from Python, which is pretty sweet. I abused the internals here, and added it to dput-ng as a quick hack. Basically, I just had to import clojure.main, and let the main function set up the sys shim. I would set it up in dput, but I d have to import that module anyway. Remember, when doing this, the namespace ((ns ...)) needs to match the filename. So, I present the first Clojure hook for dput-ng: 
; Copyright (c) Paul R. Tagliamonte <paultag@debian.org>, 2012, under the
; terms of dput-ng itsself.
(ns clojtest
  (:require dput.core
            dput.exceptions))
(defn log [x]  ; for debug output
  (.debug dput.core/logger x))
(defn dput-checker [changes profile interface]
  (cond (>= (-> changes (.get "maintainer") (.find "arno@debian.org")) 0)
    (throw (dput.exceptions/HookException. "Maintainer's Arno. Aborting upload"))
  :else
    (log "Nah, it's not arno, we're good")))
This, of course, checks if the maintainer is Arno, and throws a fit if it is. Gist is over on gist.github. After, Algernon added his own checker, which throws a fit if you have a package in the B-Ds that you aught to not use (such as dpatch) 
; Copyright (c) Gergely Nagy <algernon@debian.org>, 2012, under the
; terms of dput-ng itself.
(ns bd-blacklist
  (:require dput.core
            dput.exceptions
            dput.dsc))
(defn prune-build-deps
  "Prune a string representation of the build-depends so that only a
  list of packages remain."
  [bd-string]
  (map #(first (-> % (.strip) (.split " "))) (.. bd-string (split ","))))
(defn has-blacklisted?
  "Given a dsc file and a blacklist, check if any of the
  build-depencencies are in that list. Throws an error if there are
  matches."
  [dsc-file blacklist]
  (let [dsc (dput.dsc/parse_dsc_file dsc-file)
        build-deps (prune-build-deps (.. dsc (get "build-depends")))]
    (if-let [bad-bd (some blacklist build-deps)]
      (throw (dput.exceptions/HookException. (str "Blacklisted build-dependency found: " bad-bd)))
      (-> dput.core/logger (.trace "Build-Dependencies do not have anything on the blacklist")))))
(defn blacklist-checker
  "Checks whether the dsc has blacklisted build-dependencies, ignores
  the check when no dsc is to be found."
  [changes profile interface]
  (if-let [dsc-file (.. changes (get_dsc))]
    (has-blacklisted? dsc-file (set (get profile "bd-blacklist")))
    (-> dput.core/logger (.trace "No .dsc found, build-dependencies cannot be checked"))))
You can check that out in the git tree Hacks welcome!

16 November 2012

Paul Tagliamonte: pathfinding debian's social connections

As a followup to my toy over at debtree, I ve taken some suggestions from my buddy Arno and found some time in the hotel during a visit for work up to Canada with the Open North folks hacking on open13. The hack is called debfriends (I need to purge personal details from the source tree before I can push it to VCS, since it uses the nm.d.o dump. Sorry, I should have been better about that.) I ve posted a cute front-end on my staging machine to play with over at graph.lucifer.pault.ag There s also an API, but it has all the data in the NM dump exposed. If you re a DD who will treat this data right, I can add a key (it s not secure, like, at all). I m going to publish the source ASAP so that others can stand up an instance if they d like. It s a cute toy, though! In the tradition of using nhandler for my examples, here s an example path from pabs to Nathan: pabs > nhandler Also, we re a shockingly well connected bunch. Well done :) Feel free to play with it!

6 November 2012

Rapha&#235;l Hertzog: My Free Software Activities in October 2012

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (120.46 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg At the start of the month, I reconfigured dpkg s git repository to use KGB instead of the discontinued CIA to send out commit notices to IRC (on #debian-dpkg on OFTC, aka irc.debian.org). I didn t do anything else that affects dpkg and I must say that Guillem does not make it easy for others to get involved. He keeps all his work hidden in his private for 1.17.x branch and refuses to open an official jessie branch as can be seen from the lack of answer to this mail. On the bright side, he deals with almost all incoming bugs even before I have a chance to take care of them. But it s a pity that I can never review any of his fixes because they are usually pushed shortly before an upload. Misc packaging I helped to get #689336 fixed so that the initrd properly setups the keymap before asking for a passphrase for an encrypted partition. Related to this I filed #689722 so that cryptsetup gains a dependency ensuring that the required tools for keymap setup are available. I packaged a new upstream version of zim (0.57) and also a security update for python-django that affected both Squeeze and Wheezy. I uploaded an NMU of revelation (0.4.13-1.2) so that it doesn t get dropped from Wheezy (it was on the release team list of leaf packages that would be removed if unfixed) since my wife is using it to store her passwords. I sponsored a new upstream version of ledgersmb. Debian France We managed to elect new officers for Debian France. I m taking over the role of president, Sylveste Ledru is the new treasurer and Julien Danjou is the new secretary. Thank you very much to the former officers: Carl Chenet, Aur lien Jarno and Julien Cristau. We re in the process of managing this transition which will be completed during the next mini-Debconf in Paris so that we can exchange some papers and the like. In the first tasks that I have set myself, there s recruiting two new members for the boards of directors since we re only 7 and there are 9 seats. I made a call for volunteers and we have two volunteers. If you want to get involved and help Debian France, please candidate by answering that message as soon as possible. The Debian Handbook I merged the translations contributed on debian.weblate.org (which led me to file this wishlist bug on Weblate itself) and I fixed a number of small issues that had been reported. I made an upload to Debian to incorporate all those fixes But this is still the book covering Squeeze so I started to plan the work to update it for Wheezy and with Roland we have decided who is going to take care of updating each chapter. Librement Progress is annoyingly slow on this project. Handling money for others is highly regulated, at least in the EU apparently. I only wanted an escrow account to secure the money of users of the service but opening this account requires either to be certified as a payment institution by the Autorit de contr le prudentiel or to get an exemption from the same authority (covering only some special cases) or to sign a partnership with an established payment institution. Being certified is out of scope for now since it requires a minimum of 125000 EUR in capital (which I don t have). My bank can t sign the kind of partnership that I would need. So I have to investigate whether I can make it fit in the limited cases of exemption or I need to find another payment institution that is willing to work with me. Gittip uses Balanced a payment service specialized in market places but unfortunately it s US-only if you want to withdraw money from the system. I would love a similar service in Europe If I can t position Librement as a market place for the free software world (and save each contributor the hassle to open a merchant account), then I shall fallback to the solution where Librement only provides the infrastructure but no account, and developers who want to collect donations will have to use either Paypal or any other supported merchant account to collect funds. That s why my latest spec updates concerning the donation service and the payment service mentions Paypal and the possibility of choosing your payment service for your donation form. Thanks See you next month for a new summary of my activities.

5 comments Liked this article? Click here. My blog is Flattr-enabled.

31 October 2012

Paul Tagliamonte: dput-ng -- the next generation of your friendly upload tool

Hello, World! Arno and I have been in the middle of a 3 week long quest to rewrite dput. We feature tons of improvements over the old dput, such as: Support for even core features are treated as plugins, so derivative distributions should be able to add features without maintaining a patch against dput (current situation). In addition, I ve implemented some show-ey post-upload hooks, such as a Twitter plugin. Here s how it s defined: 
import twitter
import json
import os
def tweet(changes, profile, interface):
    tweet = "I've just uploaded %s/%s to %s's %s suite #debian" % (
        changes['Source'],
        changes['Version'],
        profile['name'],
        changes['Distribution']
    )
    if len(tweet) > 140:
        tweet = tweet[:140]
    obj = json.load(open(os.path.expanduser("~/.twitter.json"), 'r'))
    t = twitter.Api(
        consumer_key=obj['consumer_key'],
        consumer_secret=obj['consumer_secret'],
        access_token_key=obj['oath_token'],
        access_token_secret=obj['oath_secret']
    )
    t.PostUpdates(tweet)
This file is placed in ~/.dput.d/scripts, as tweet.py, and the plugin def file is placed in ~/.dput.d/processors (since it s a post-upload processor), which looks like: 
 
    "name": "Tweet after an upload.",
    "path": "tweet.tweet"
I know this sounds confusing, but it s actually quite nice. Check out the plugin in action Feedback, and contributors more then welcome! Documentation is at dput.rtfd.org. It s all a bit pre-beta at the moment, but the authors are using it for all their uploads, and should be stable for users who don t mind a bit of breakage. We ve filed the ITP, and updates on it & and upload will follow shortly. We re just weeding out some final major issues. Should be stable enough for general use in the next 30 days or so.

19 October 2012

Martin F. Krafft: Configuration management

Puppet I've really had it with Puppet. I used to be able to put up with all its downsides
  • Non-Unix approach to everything (own transport, self-made PKI, non-intuitive configuration language, a faint attempt at versioning (bitbucket), and much much more )
  • Ruby
  • Abysmal slowness
  • Lack of basic functionality (e.g. replace a line of text)
  • Host management and configuration programming intertwined, lack of a high-level approach to defining functionality
  • Horrific error messages
  • Catastrophic upgrade paths
  • Did I mention Ruby and its speed?
  • Lack of IPv6 support
  • [I could keep going ]
but now that my fourth attempt to upgrade my complex configuration from version 0.25.5 to version 2.7 failed due to a myriad of completely incomprehensible errors ("err: Could not run Puppet configuration client: interning empty string") and many hours were lost in trying to hunt these down using binary searches, I am giving up. Bye bye Puppet.

An alternative But I need an alternative. I want a system that is capable of handling a large number of hosts, but not so complex that one wouldn't put it to use for half a dozen machines. The configuration management system I want looks about as follows: It
  • makes use of existing infrastructure (e.g. SSH transport and public keys, Unix toolchain, Debian package management and debconf)
  • interacts with the package management system (Debian only in my case)
  • can provision files whose contents might depend on context, particular machine data and conditionals. There should be a unified templating approach for static and dynamic files, with the ability to override the source of data (e.g. a default template used unless a template exists for a class of machine, or a specific hostname)
  • can edit files on the target machine in a flexible and robust manner
  • can remove files
  • can run commands when files change
  • can reference data from other machines (e.g. obtain the certificate fingerprint of each hosts that define me as their SMTP smarthost)
  • can control running services (i.e. enable init.d scripts, check that a process is running
  • is written in a sensible language
  • is modular and easily extensible, ideally using a well-known language (e.g. Python!)
  • allows to specify infrastructure with tags ("all webservers", "all machines in Zurich", "machines that are in Munich and receive mail"), but with the ability to override every parameter for a specific host
  • should just do configuration management, and not try to take away jobs from monitoring software
  • logs changes per-machine and collects data about applied configurations in a central location
  • is configured using flat files that are human-readable so that the configuration may be stored in Git (e.g. YAML, not XML)
  • can be configured using scripts in a flexible way
Since for me, Ruby is a downside of Puppet, I won't look at Chef, but from this page, I gleaned a couple of links: Ansible, Quattor, Salt, and bcfg2 (which uses XML though). And of course, there remains the ephemeral cfengine.

cfengine I haven't used cfengine since 2002, but I am not convinced it's worth a new look because it seems to be an academic project with gigantic complexity and a whole vernacular to its own. There is no doubt that it is a powerful solution, and the most mature of all of them, but it's far away from the Unix-like simplicity that I've come to love in almost 20 years of Debian. Do correct me if I am wrong.

Ansible Ansible looks interesting. It seems rather bottom-up, first introducing a way to remotely execute commands on hosts, which you can then later extend/automate to manage the host configurations. It uses SSH for transport, and its reason-to-be made me want to look at it. My ventures into the Ansible domain are not over yet, but I've put them on hold. First of all, it's not yet packaged for Debian (Ubuntu-PPA packages work on Debian squeeze and wheezy). Second, I was put off a bit by its gratuitous use of the shell to run commands, as well as other design decisions. Check this out: there are modules for the remote execution of commands, namely "shell", "command", and "raw". The shell modules should be self-explanatory; the command module provides some idempotency, such as not running the command if a file exists (or not). To do this, it creates a Python script in /tmp on the target and then executes that like so: 
$SHELL -c /tmp/ansible/ansible-1350291485.22-74945524909437/command
Correct me if I am wrong, but there is zero need for this shell indirection. My attempts at finding an answer on IRC were met by user "daniel_hozac" with a reason along the lines of "it's needed, believe me", and on the mailing list, I am told that only the shell can execute a script by parsing the interpreter line at the top of the module. Finally, the raw execution module also executes using the shell And there a few other design decisions that I can't quite explain, around the command-line switch --sudo see the aforementioned message In short: running a command like 
ansible -v arnold.madduck.net -a "/usr/bin/apt-get update" --sudo
does not invoke apt-get with sudo, as one might like; it invokes the shell that runs the Python script that runs the command. Effectively therefore, you need to allow sudo shell execution, and for proper automation, this has to be possible without a password. And then you might just as well allow root logins again. The author seems to think that "core behaviour" is that sudo allows all execution and that limiting the commands to run is not a use-case that Ansible will support. Apparently, I was the first to ever suggest this. There are always ways around (e.g. skip --sudo and just use sudo as the command, simply ignore the useless shell invocation and trust that your machine can handle it, but when such design decisions remain incomprehensible and get defended by the project people, then I am hesitant to invest more time on principle.

Salt Finally, I've looked at Salt, which is what I've spent most time on so far. From the discussions I started on host targeting and data collection, it soon became apparent that Salt is very thin and flexible, and that the user community is accomodating. Unfortunately, Salt does not use SSH, but at least it reuses existing functionality (ZeroMQ). As opposed to the push/pull model, Salt "minions" interestingly maintain a persistent connection to the server (which is not yet very stable), and while non-root usage is still not unproblematic, at least there has already been work done in this direction. I think I will investigate Salt more as it does look like it can do what I want. The YAML-based syntax does seem a bit brittle, but it's the best I've found so far. NP: The Pineapple Thief: Someone Here is Missing

14 October 2012

Arnaud Quette: Cryptography: SSL support using Mozilla NSS landed in NUT

this week, a new feature was merged into the NUT development tree: beside of the historical OpenSSL support (for more than 10 years), NUT now also provides SSL features using Mozilla NSS. I've already posted a lengthy mail on the NUT developers list. But there are still a few things to be told: For the braves who want to test, here is a small procedure, adapted to Debian. We will use an existing installation, and overwrite upsd, upsmon, libupsclient and few more. Beware that it will obviously breaks the MD5 sums of your nut packages! 
# apt-get install nut

$ tar xzvf nut-trunk-r3751.tar.gz

# apt-get install libnss3-dev 

./configure --without-all --with-nss \
        --prefix=/ --sysconfdir=/etc/nut  \
        --with-statepath=/var/run/nut \
        --with-altpidpath=/var/run/nut \
        --with-drvpath=/lib/nut \
        --with-pidpath=/var/run/nut \
        --datadir=/usr/share/nut \
        --with-pkgconfig-dir=/usr/lib/pkgconfig \
        --with-user=nut --with-group=nut

$ make && make install
Before concluding, here is the traditional thank you guys: As a conclusion, cryptography integration and usability is still not on par with other proprietary OS. I would love to see the crypto situation improving in Debian (and friends obviously). So this was just my 2 cents (nuts ;-)) to the cause... cheers,
-- Arno

25 September 2012

Arnaud Quette: Power management and NUT #1: an introduction

in this series of articles, I will be talking in depth about power management through the NUT project, its packaging on Debian, how to use it in general and how I see it being part of the GreenIT thing. So, let's start with an introduction: NUT is a Free Software (GPL v2+ and 3 to be precise), originally created for power protection using UPS, from home to data-centers:
To shortly describe the main features, I would say that NUT: NUT used to stand for Network UPS Tools. That is, a software for talking to your UPS and shutting down your systems when needed. This definition is a bit limited nowadays, since NUT supports 4 types of power device: Considering this, is the name Network UPS Tools still suitable? Not really! But the acronym NUT is well known! So, for the time being, I just stick using it, and focus on other more important things, until a better opportunity (ideas and comments are welcome!). That said, what can you do exactly with NUT? Currently, you can: All the above is available in a standardized way: Well, this is already a long and dense post, so I will stop there for today. In the next post, we will have a deeper dive into using NUT, for various use cases: submit yours if you can ;-) cheers,
-- Arno

24 August 2012

Olivier Berger: Generating RDF description of Debian package sources with ADMS.SW

Edit : I ve now managed to roll out my contribution which is now in production on packages.qa.debian.org. See a later post I ve made on the subject, and beware that the generated RDF has changed a bit also. ADMS.SW proposes specifications for description of software present in software catalogues. I ve tried and apply it to the contents of the Debian Package Tracking System (PTS), using transformation of the information known by the PTS to RDF+XML. The result is not yet in production, but here s an example of what can be done, using the Turtle syntax (more readable) :
<style type="text/css"> .T1 color:#000000; font-size:14pt; font-weight:bold; .T2 color:#000000; font-size:12pt; .T3 color:#000000; font-size:8.5pt; .T4 color:#0000ff; font-size:8.5pt; .T5 color:#a020f0; font-size:8.5pt; .T6 color:#228b22; font-size:8.5pt; .T7 color:#b22222; font-size:8.5pt; .T8 color:#008b8b; font-size:8.5pt; .T9 color:#8b2252; font-size:8.5pt; .dp1 .dp2 </style> @base <http://packages.qa.debian.org/> .
@prefix rdf: <http://www.w3.org/1999/02/22 rdf syntax ns#> .
@prefix admssw: <http://purl.org/adms/sw/> .
@prefix doap: <http://usefulinc.com/ns/doap#> .
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf schema#> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix schema: <http://schema.org/> .
@prefix spdx: <http://www.spdx.org/rdf/terms#> .
@prefix : <http://www.w3.org/1999/xhtml> .
@prefix str: <http://exslt.org/strings> . #
# First the packaging project for apache2 in Debian
#
# <http://packages.qa.debian.org/apache2> resource :
<apache2>
a admssw:SoftwareProject ;
doap:name "apache2" ;
doap:description "Debian apache2 source packaging" ;
doap:homepage "http://packages.debian.org/src:apache2" ;
schema:contributor [
a foaf:OnlineAccount ;
foaf:accountName "Debian Apache Maintainers" ;
foaf:accountServiceHomepage <http://qa.debian.org/developer.php?login=debian apache@lists.debian.org>
], [
a foaf:OnlineAccount ;
foaf:accountName "Stefan Fritsch" ;
foaf:accountServiceHomepage <http://qa.debian.org/developer.php?login=sf@debian.org>
], [
a foaf:OnlineAccount ;
foaf:accountName "Steinar H. Gunderson" ;
foaf:accountServiceHomepage <http://qa.debian.org/developer.php?login=sesse@debian.org>
], [
a foaf:OnlineAccount ;
foaf:accountName "Arno T ll" ;
foaf:accountServiceHomepage <http://qa.debian.org/developer.php?login=arno@debian.org>
] ;
# pointer to the release in the different suites :
doap:release <apache2_2.2.16 6+squeeze7>, <apache2_2.2.22 11>, <apache2_2.4.2 2>. #
# Now the different debian package source releases
#
# <http://packages.qa.debian.org/apache2_2.2.16 6+squeeze7> resource
<apache2_2.2.16 6+squeeze7>
a admssw:SoftwareRelease ;
rdfs:label "apache2 2.2.16 6+squeeze7" ;
admssw:project <apache2> ;
dcterms:description "Debian apache2 source package version 2.2.16 6+squeeze7" ;
doap:revision "2.2.16 6+squeeze7" . # This one is the reference version for the PTS as in unstable, so
# contains more details than the others
<apache2_2.2.22 11>
a admssw:SoftwareRelease ;
rdfs:label "apache2 2.2.22 11" ;
admssw:project <apache2> ;
dcterms:description "Debian apache2 source package version 2.2.22 11" ;
doap:revision "2.2.22 11" ;
# this release contains two components
admssw:includedAsset <apache2/apache2_2.2.22 11_debian>, <apache2/apache2_2.2.22_orig>
;
# this release can be downloaded as one package (with dget)
admssw:package <apache2/apache2_2.2.22 11.dsc> ;
# it also has a related release somewhere else
dcterms:relation <https://launchpad.net/ubuntu/+source/apache2/2.2.22 6ubuntu2> .
<apache2_2.4.2 2>
a admssw:SoftwareRelease ;
rdfs:label "apache2 2.4.2 2" ;
dcterms:description "Debian apache2 source package version 2.4.2 2" ; admssw:project <apache2> ;
doap:revision "2.4.2 2" . # Then the .dsc file for the current unstable version
<apache2/apache2_2.2.22 11.dsc>
a admssw:SoftwarePackage ;
dcterms:description "Debian source package descriptor file for apache2 version 2.2.22 11";
schema:downloadUrl "http://cdn.debian.net/debian/pool/main/a/apache2/apache2_2.2.22 11.dsc";
schema:fileSize "2885" ;
spdx:checksum [
a spdx:Checksum ;
spdx:algorithm <apache2#checksumAlgorithm_md5sum> ;
spdx:checksumValue "d7d03719b9f6432beeecd3aa04f7b22c"
] . #
# Then the upstream project
#
<apache2/apache2_orig>
a admssw:SoftwareProject ;
doap:description "The apache2 upstream project" ;
# either its name or homepage can be matched against other ADMS.SW
# or DOAP descriptors
doap:name "apache2" ;
doap:homepage "http://httpd.apache.org/" . # And a upstream release 2.2.22
<apache2/apache2_2.2.22_orig>
a admssw:SoftwareRelease ;
rdfs:label "Upstream apache2 release 2.2.22" ;
dcterms:description "Upstream source release for apache2 version 2.2.22" ;
doap:revision "2.2.22" ;
admssw:project <apache2/apache2_orig> ;
# and a link to the upstream source tarball s description
admssw:package <apache2/apache2_2.2.22.orig.tar.gz> . # now the (potentially re archived) upstream source archive for that release
<apache2/apache2_2.2.22.orig.tar.gz>
a admssw:SoftwarePackage ;
dcterms:description "Upstream source archive for apache2 version 2.2.22 11 (potentially re archived by Debian)";
schema:downloadUrl "http://cdn.debian.net/debian/pool/main/a/apache2/apache2_2.2.22.orig.tar.gz";
# Those 2 bits should help match packagings of the same tarball if needed
schema:fileSize "7200529" ;
spdx:checksum [
a spdx:Checksum ;
spdx:algorithm <apache2#checksumAlgorithm_md5sum> ;
spdx:checksumValue "d77fa5af23df96a8af68ea8114fa6ce1"
] . #
# Now, document specific details of the second component : Debian
# packaging files archive
#
<apache2/apache2_2.2.22 11_debian>
a admssw:SoftwareRelease ;
rdfs:label "Package sources apache2_2.2.22 11" ;
dcterms:description "Debian packaging sources for apache2 version 2.2.22 11" ;
doap:revision "2.2.22 11" ;
admssw:package <apache2/apache2_2.2.22 11.debian.tar.gz> . <apache2/apache2_2.2.22 11.debian.tar.gz>
a admssw:SoftwarePackage ;
dcterms:description "Debian source package files archive for apache2 version 2.2.22 11" ;
schema:downloadUrl "http://cdn.debian.net/debian/pool/main/a/apache2/apache2_2.2.22 11.debian.tar.gz";
schema:fileSize "195980" ;
spdx:checksum [
a spdx:Checksum ;
spdx:algorithm <apache2#checksumAlgorithm_md5sum> ;
spdx:checksumValue "d3d4146ccad51129636b7dbff284a110"
] . #
# Now, we also weave links with the Ubuntu counterparts
#
<https://launchpad.net/ubuntu/+source/apache2>
a admssw:SoftwareProject ;
doap:description "\"apache2\" source package in Ubuntu" ;
# this one promises some trolls and flames
admssw:forkOf <apache2> ;
doap:homepage "https://launchpad.net/ubuntu/+source/apache2" ;
doap:release <https://launchpad.net/ubuntu/+source/apache2/2.2.22 6ubuntu2> . # and its release known by the PTS
<https://launchpad.net/ubuntu/+source/apache2/2.2.22 6ubuntu2>
a admssw:SoftwareRelease ;
rdfs:label "apache2 2.2.22 6ubuntu2" ;
dcterms:description "\"apache2\" 2.2.22 6ubuntu2 source package in Ubuntu" ;
doap:revision "2.2.22 6ubuntu2" ;
admssw:project <https://launchpad.net/ubuntu/+source/apache2> .

Next.

Previous.